1. Field of the Invention
This invention relates to a light source direction estimating method and apparatus, wherein a direction of a light source is estimated from an image having been obtained by performing an image recording operation by use of a single light source or a plurality of light sources. This invention also relates to a recording medium, on which a program for causing a computer to execute the light source direction estimating method has been recorded and from which the computer is capable of reading the program.
2. Description of the Related Art
Techniques for combining a real image and a virtual image, such as a computer graphics (CG) image, to obtain a composed image giving no unnatural feeling are referred to as mixed reality techniques and are expected to play an important role in various fields, such as communication, traffic, and amusements. In order for a real image and a virtual image to be combined such that a composed image giving no unnatural feeling can be obtained, it is necessary for the real image and the virtual image to be combined with geometrical, optical, and temporal consistency. In particular, as for the optical consistency, if the direction of a light source and the intensity of light radiated out from the light source with respect to the real image and the direction of a light source and the intensity of light radiated out from the light source with respect to the virtual image do not coincide with each other, a composed image giving an unnatural feeling is obtained. Therefore, the optical consistency is important markedly. Accordingly, a technique for estimating a distribution of intensity of light radiated out from a light source in accordance with an image has been proposed in, for example, xe2x80x9cEstimation of illumination distribution by using soft shadowsxe2x80x9d by I. Sato, et al., Computer Vision And Image Media, 110-3, Mar. 19, 1998, pp. 17-24.
With the technique proposed by Sato, et al., an intensity distribution of environmental illumination is estimated on the basis of a soft shadow of a real object. Specifically, the intensity distribution of environmental illumination is estimated by utilizing a shadow of an object, whose three-dimensional shape is known, the shadow appearing in an image. More specifically, relationship between brightness of a virtual light source and a signal value at a certain measuring point on the object is expressed with a determinant, an inverse matrix of the matrix in the determinant is calculated, and the intensity distribution of light radiated out from the light source is thereby estimated. In cases where the intensity distribution of light radiated out from the light source is estimated from the image with the proposed technique, the intensity distribution of light radiated out from a light source with respect to a virtual image can be set to be identical with the intensity distribution of light radiated out from the light source with respect to the real image, and therefore the real image and the virtual image can be combined to form a composed image giving no unnatural feeling.
However, with the technique proposed by Sato, et al., it is necessary for the position of the measuring point to be set at a position which is mathematically linearly independent and which allows calculation of the inverse matrix. It is necessary for the position of the measuring point to be found with manual operations. Therefore, the operations cannot be kept simple.
The primary object of the present invention is to provide a light source direction estimating method, with which a direction of a single light source is capable of being estimated easily.
Another object of the present invention is to provide a light source direction estimating method, with which directions of a plurality of light sources are capable of being estimated easily.
A further object of the present invention is to provide an apparatus for carrying out the light source direction estimating method.
A still further object of the present invention is to provide a recording medium, on which a program for causing a computer to execute the light source direction estimating method has been recorded and from which the computer is capable of reading the program.
The present invention provides a first light source direction estimating method, wherein an image is obtained, which image has been formed by performing an image recording operation on an object under a substantially single light source, the object containing an area, which has approximately uniform surface reflectivity and which exhibits diffuse reflection, and wherein a direction of the light source, under which the image recording operation was performed, is estimated in accordance with the image, the method comprising the step of:
estimating the direction of the light source., under which the image recording operation was performed, in accordance with signal values of an object image signal, which represents an object image embedded in the image having been obtained from the image recording operation, and relationship between a position of a three-dimensional body image, which is represented by a position-matched three-dimensional shape signal that has been obtained from position matching performed on the object image signal and a three-dimensional shape signal representing a three-dimensional shape of the object, and positions of virtual light sources having been set in a plurality of arbitrary directions.
The term xe2x80x9cperforming an image recording operation under a substantially single light sourcexe2x80x9d as used herein means that the image recording operation is performed in a state approximately identical to the state in which it can be regarded that a single light source alone is utilized and the presence of other light sources can be ignored as in the cases where the image recording operation is performed under the sunlight in fine weather.
The term xe2x80x9cthree-dimensional shape signal representing a three-dimensional shape of an objectxe2x80x9d as used herein means the signal representing the shape of the object itself.
The term xe2x80x9cposition matching performed on an object image signal and a three-dimensional shape signalxe2x80x9d as used herein means the operation for matching the object image signal and the three-dimensional shape signal such that the orientation, the center position, the size, and the like, of the object in the image, which has been obtained from the image recording operation, and the orientation, the center position, the size, and the like, of the three-dimensional shape of the object may coincide with each other.
The first light source direction estimating method in accordance with the present invention should preferably be modified such that a plurality of measuring points are set at an area of the three-dimensional body image, which is represented by the position-matched three-dimensional shape signal that has been obtained from the position matching, the area of the three-dimensional body image corresponding to the area of the object, which area has approximately uniform surface reflectivity and which exhibits diffuse reflection,
a calculation is made to find a cosine between a direction vector directed from each of virtual light sources, which have been set in a plurality of arbitrary directions, to each of the measuring points and a normal vector at each of the measuring points,
a relationship between the calculated cosines and signal values of the object image signal, which signal values correspond to the plurality of the measuring points, is plotted with respect to each of the directions of the virtual light sources, and
a direction of a virtual light source, which direction is associated with a plotted relationship that forms an approximately straight line, is estimated as the direction of the light source, under which the image recording operation was performed.
The direction vector can be calculated from the spatial coordinates of the virtual light source and the spatial coordinates of the measuring point. The normal vector can be calculated from the signal value of the position-matched three-dimensional shape signal, which signal value corresponds to the spatial coordinates of the measuring point. Also, the cosine can be calculated from the inner product of the direction vector and the normal vector.
Also, the first light source direction estimating method in accordance with the present invention should preferably be modified such that a judgment as to whether the plotted relationship forms or does not form an approximately straight line is made in accordance with an evaluating function utilizing a method of least squares, and the estimation of the direction of the light source, under which the image recording operation was performed, is performed in accordance with the results of the judgment.
Further, the first light source direction estimating method in accordance with the present invention should preferably be modified such that intensity of light radiated out from the light source, under which the image recording operation was performed, is estimated in accordance with an inclination of a straight line that is set by the plotted relationship utilized for the estimation of the direction of the light source, under which the image recording operation was performed.
Furthermore, in the first light source direction estimating method in accordance with the present invention, the object should preferably be a face of a person.
A flesh color area of a face of a person can be regarded as having approximately uniform surface reflectivity.
Also, in the first light source direction estimating method in accordance with the present invention, the three-dimensional shape signal should preferably be a polygon signal representing the three-dimensional shape of the object.
The present invention also provides a first light source direction estimating apparatus, wherein an image is obtained, which image has been formed by performing an image recording operation on an object under a substantially single light source, the object containing an area, which has approximately uniform surface reflectivity and which exhibits diffuse reflection, and wherein a direction of the light source, under which the image recording operation was performed, is estimated in accordance with the image, the apparatus comprising:
means for estimating the direction of the light source, under which the image recording operation was performed, in accordance with signal values of an object image signal, which represents an object image embedded in the image having been obtained from the image recording operation, and relationship between a position of a three-dimensional body image, which is represented by a position-matched three-dimensional shape signal that has been obtained from position matching performed on the object image signal and a three-dimensional shape signal representing a three-dimensional shape of the object, and positions of virtual light sources having been set in a plurality of arbitrary directions.
The first light source direction estimating apparatus in accordance with the present invention should preferably be modified such that the means for estimating the direction of the light source, under which the image recording operation was performed, comprises:
position matching means for performing position matching on the object image signal, which represents the object image embedded in the image having been obtained from the image recording operation, and the three-dimensional shape signal representing the three-dimensional shape of the object,
measuring point setting means for setting a plurality of measuring points at an area of the three-dimensional body image, which is represented by the position-matched three-dimensional shape signal that has been obtained from the position matching, the area of the three-dimensional body image corresponding to the area of the object, which area has approximately uniform surface reflectivity and which exhibits diffuse reflection,
cosine calculating means for calculating a cosine between a direction vector directed from each of virtual light sources, which have been set in a plurality of arbitrary directions, to each of the measuring points and a normal vector at each of the measuring points,
plotting means for plotting a relationship between the calculated cosines and signal values of the object image signal, which signal values correspond to the plurality of the measuring points, with respect to each of the directions of the virtual light sources, and
direction estimating means for estimating a direction of a virtual light source, which direction is associated with a plotted relationship that forms an approximately straight line, as the direction of the light source, under which the image recording operation was performed.
Also, the first light source direction estimating apparatus in accordance with the present invention should preferably be modified such that the direction estimating means comprises judgment means for making a judgment as to whether the plotted relationship forms or does not form an approximately straight line, the judgment being made in accordance with an evaluating function utilizing a method of least squares, and performing the estimation of the direction of the light source, under which the image recording operation was performed, in accordance with the results of the judgment.
Further, the first light source direction estimating apparatus in accordance with the present invention should preferably be modified such that the apparatus further comprises intensity estimating means for estimating intensity of light radiated out from the light source, under which the image recording operation was performed, in accordance with an inclination of a straight line that is set by the plotted relationship utilized for the estimation of the direction of the light source, under which the image recording operation was performed.
Furthermore, in the first light source direction estimating apparatus in accordance with the present invention, the object should preferably be a face of a person.
Also, in the first light source direction estimating apparatus in accordance with the present invention, the three-dimensional shape signal should preferably be a polygon signal representing the three-dimensional shape of the object.
The present invention further provides a recording medium, on which a program for causing a computer to execute the first light source direction estimating method in accordance with the present invention has been recorded and from which the computer is capable of reading the program.
The present invention still further provides a second light source direction estimating method, wherein an image is obtained, which image has been formed by performing an image recording operation on an object under a plurality of light sources such that light interception occurs, the object containing an area, which has approximately uniform surface reflectivity and which exhibits diffuse reflection, and wherein a direction of each of the light sources, under which the image recording operation was performed, is estimated in accordance with the image, the method comprising the step of:
estimating the direction of each of the light sources, under which the image recording operation was performed, in accordance with signal values of an object image signal, which represents an object image embedded in the image having been obtained from the image recording operation, and relationship between a position of a three-dimensional body image, which is represented by a position-matched three-dimensional shape signal that has been obtained from position matching performed on the object image signal and a three-dimensional shape signal representing a three-dimensional shape of the object, and positions of virtual light sources having been set in a plurality of arbitrary directions.
The object employed in the second light source direction estimating method in accordance with the present invention is an object which has unevenness on its surface and whose three-dimensional shape is known.
In the second light source direction estimating method in accordance with the present invention, with the position matching performed on the object image signal and the three-dimensional shape signal, the object image signal and the three-dimensional shape signal are matched with each other such that the orientation, the center position, the size, and the like, of the object in the image, which has been obtained from the image recording operation, and the orientation, the center position, the size, and the like, of the three-dimensional shape of the object may coincide with each other.
The second light source direction estimating method in accordance with the present invention should preferably be modified such that the direction of each of the light sources, under which the image recording operation was performed, is estimated by:
selecting combinations of virtual light sources from the virtual light sources having been set in the plurality of the arbitrary directions,
setting a plurality of measuring points at an area of the three-dimensional body image, which is represented by the position-matched three-dimensional shape signal that has been obtained from the position matching, the area of the three-dimensional body image corresponding to the area of the object, which area has approximately uniform surface reflectivity and which exhibits diffuse reflection,
selecting single-irradiation measuring points, which are exposed to only light radiated out from a single virtual light source when light is irradiated from each of the combinations of the virtual light sources to the three-dimensional body image, from the plurality of the measuring points and in accordance with the object image signal,
calculating a cosine between a direction vector, which is directed from the single virtual light source to each of the single-irradiation measuring points, and a normal vector at each of the single-irradiation measuring point,
plotting a relationship between the calculated cosines and signal values of the object image signal, which signal values correspond to the single-irradiation measuring points, with respect to each of the directions of the virtual light sources,
estimating a direction of a virtual light source, which direction is associated with a plotted relationship that forms an approximately straight line, as a direction of one light source among the light sources, under which the image recording operation was performed,
estimating a relative intensity value of light radiated out from the one light source in accordance with an inclination of a straight line, which is set by the plotted relationship utilized for the estimation of the direction of the one light source,
removing effects of light, which is radiated out from the one light source, from the object image signal, which has been subjected to the position matching, in accordance with the relative intensity value of light radiated out from the one light source, to acquire a corrected object image signal, and
iterating the selection of combinations of virtual light sources from the virtual light sources other than the virtual light source located in the direction of the one light source, the selection of single-irradiation measuring points in accordance with the corrected object image signal being employed as a new object image signal, the calculation of cosines, the plotting, the estimation of a direction of one light source, the estimation of a relative intensity value of light radiated out from the one light source, and the acquisition of a corrected object image signal, till directions of all of the light sources, under which the image recording operation was performed, are estimated.
In cases where, for example, the object has a convex part, an area which is hidden behind the convex part and upon which light does not impinge occurs, depending upon the direction of a light source. In this specification, the state in which light does not impinge upon an area is referred to as the light interception.
The term xe2x80x9cselecting a combination of virtual light sources from virtual light sources having been set in a plurality of arbitrary directionsxe2x80x9d as used herein means that a combination of a certain number of virtual light source is selected from the virtual light sources having been set in the plurality of the arbitrary directions. Specifically, in cases where the number of the virtual light sources having been set is represented by n, and the number of the virtual light sources constituting each combination is represented by m, where m=1 to n, the total number of the combinations is equal to the sum of combinations nCm for m=1 to n.
The term xe2x80x9csingle-irradiation measuring pointxe2x80x9d as used herein means the point such that, when calculations are made to find vectors, each of which is directed from each of the virtual light sources constituting a combination to each of the measuring points having been set at an area of the three-dimensional body image represented by the position-matched three-dimensional shape signal that has been obtained from the position matching, only a single vector among the thus calculated vectors can reach the point without being intercepted by the three-dimensional body image due to the light interception or the cosine taking a negative value.
The direction vector can be calculated from the spatial coordinates of the virtual light source and the spatial coordinates of the single-irradiation measuring point. The normal vector can be calculated from the signal value of the position-matched three-dimensional shape signal, which signal value corresponds to the spatial coordinates of the measuring point. Also, the cosine can be calculated from the inner product of the direction vector and the normal vector.
The object image signal is obtained as intensity values of light reflected by the object when the light radiated out from the plurality of the light sources impinges upon the object. The intensity values obtained with the light, which is radiated out from the one light source having been estimated, represent the image of the object obtained when only the light radiated out from the light source located in the estimated direction impinges upon the object. Therefore, in cases where the intensity values obtained with the light, which is radiated out from the one light source having been estimated, are removed from the object image signal, the intensity values, which occur when the light having been radiated out from the light sources other than the one light source having been estimated is reflected from the object, can be obtained. The thus obtained intensity values are taken as the corrected object image signal.
Also, the second light source direction estimating method in accordance with the present invention should preferably be modified such that, in cases where the number of the light sources, under which the image recording operation was performed, is known, combinations of the virtual light sources in accordance with the number of the light sources, under which the image recording operation was performed, are selected from the virtual light sources having been set in the plurality of the arbitrary directions.
In cases where the number of the virtual light sources having been set in the plurality of the arbitrary directions is represented by n, and the number of the light sources, under which the image recording operation was performed, is represented by k, the number of the combinations of the virtual light sources becomes equal to nCk.
Further, the second light source direction estimating method in accordance with the present invention should preferably be modified such that a judgment as to whether the plotted relationship forms or does not form an approximately straight line is made in accordance with an evaluating function utilizing a method of least squares, and the estimation of the direction of the light source is performed in accordance with the results of the judgment.
Furthermore, in the second light source direction estimating method in accordance with the present invention, the object should preferably be a face of a person.
A flesh color area of a face of a person can be regarded as having approximately uniform surface reflectivity.
Also, in the second light source direction estimating method in accordance with the present invention, the three-dimensional shape signal should preferably be a polygon signal representing the three-dimensional shape of the object.
The present invention also provides a second light source direction estimating apparatus, wherein an image is obtained, which image has been formed by performing an image recording operation on an object under a plurality of light sources such that light interception occurs, the object containing an area, which has approximately uniform surface reflectivity and which exhibits diffuse reflection, and wherein a direction of each of the light sources, under which the image recording operation was performed, is estimated in accordance with the image, the apparatus comprising:
means for estimating the direction of each of the light sources, under which the image recording operation was performed, in accordance with signal values of an object image signal, which represents an object image embedded in the image having been obtained from the image recording operation, and relationship between a position of a three-dimensional body image, which is represented by a position-matched three-dimensional shape signal that has been obtained from position matching performed on the object image signal and a three-dimensional shape signal representing a three-dimensional shape of the object, and positions of virtual light sources having been set in a plurality of arbitrary directions.
The second light source direction estimating apparatus in accordance with the present invention should preferably be modified such that the means for estimating the direction of each of the light sources, under which the image recording operation was performed, comprises:
combination selecting means for selecting combinations of virtual light sources from the virtual light sources having been set in the plurality of the arbitrary directions,
measuring point setting means for setting a plurality of measuring points at an area of the three-dimensional body image, which is represented by the position-matched three-dimensional shape signal that has been obtained from the position matching, the area of the three-dimensional body image corresponding to the area of the object, which area has approximately uniform surface reflectivity and which exhibits diffuse reflection,
single-irradiation measuring point selecting means for selecting single-irradiation measuring points, which are exposed to only light radiated out from a single virtual light source when light is irradiated from each of the combinations of the virtual light sources to the three-dimensional body image, from the plurality of the measuring points and in accordance with the object image signal,
cosine calculating means for calculating a cosine between a direction vector, which is directed from the single virtual light source to each of the single-irradiation measuring points, and a normal vector at each of the single-irradiation measuring point,
plotting means for plotting a relationship between the calculated cosines and signal values of the object image signal, which signal values correspond to the single-irradiation measuring points, with respect to each of the directions of the virtual light sources,
direction estimating means for estimating a direction of a virtual light source, which direction is associated with a plotted relationship that forms an approximately straight line, as a direction of one light source among the light sources, under which the image recording operation was performed,
intensity value estimating means for estimating a relative intensity value of light radiated out from the one light source in accordance with an inclination of a straight line, which is set by the plotted relationship utilized for the estimation of the direction of the one light source,
correction means for removing effects of light, which is radiated out from the one light source, from the object image signal, which has been subjected to the position matching, in accordance with the relative intensity value of light radiated out from the one light source, to acquire a corrected object image signal, and
control means for controlling the combination selecting means, the measuring point setting means, the single-irradiation measuring point selecting means, the cosine calculating means, the plotting means, the direction estimating means, the intensity value estimating means, and the correction means, such that the selection of combinations of virtual light sources from the virtual light sources other than the virtual light source located in the direction of the one light source, the selection of single-irradiation measuring points in accordance with the corrected object image signal being employed as a new object image signal, the calculation of cosines, the plotting, the estimation of a direction of one light source, the estimation of a relative intensity value of light radiated out from the one light source, and the acquisition of a corrected object image signal, are iterated till directions of all of the light sources, under which the image recording operation was performed, are estimated.
Also, the second light source direction estimating apparatus in accordance with the present invention should preferably be modified such that, in cases where the number of the light sources, under which the image recording operation was performed, is known, the combination selecting means selects combinations of the virtual light sources in accordance with the number of the light sources, under which the image recording operation was performed, from the virtual light sources having been set in the plurality of the arbitrary directions.
Further, the second light source direction estimating apparatus in accordance with the present invention should preferably be modified such that the direction estimating means comprises judgment means for making a judgment as to whether the plotted relationship forms or does not form an approximately straight line, the judgment being made in accordance with an evaluating function utilizing a method of least squares, and performing the estimation of the direction of the light source in accordance with the results of the judgment.
Furthermore, in the second light source direction estimating apparatus in accordance with the present invention, the object should preferably be a face of a person.
Also, in the second light source direction estimating apparatus in accordance with the present invention, the three-dimensional shape signal should preferably be a polygon signal representing the three-dimensional shape of the object.
The present invention further provides a recording medium, on which a program for causing a computer to execute the second light source direction estimating method in accordance with the present invention has been recorded and from which the computer is capable of reading the program.
With the first light source direction estimating method and apparatus in accordance with the present invention, in cases where only the signal values of the object image signal and the relationship between the position of the three-dimensional body image, which is represented by the position-matched three-dimensional shape signal that has been obtained from the position matching with the object image signal, and the positions of virtual light sources having been set in the plurality of the arbitrary directions are found, the direction of the light source, under which the image recording operation was performed, can be estimated. Therefore, it is not necessary for a measuring point, which allows calculation of an inverse matrix, to be found as in the technique proposed by Sato, et al. Also, it is not necessary for matrix operations to be performed. Accordingly, the direction of the light source, under which the image recording operation was performed, can be estimated with comparatively simple operations. Further, in cases where a real image and a virtual image, such as a CG image, are to be combined with each other, the virtual image can be corrected in accordance with the direction of the light source having been estimated from the real image, such that the virtual image is illuminated by an appropriate light source, and the real image and the virtual image can then be combined with each other. Therefore, a composed image giving no unnatural feeling can be obtained. Furthermore, in an encoding system, such as MPEG-4, wherein an image is transferred with respect to each of objects and image composition is performed at the stage of seeing an animation, in cases where a real image and a virtual image are contained in the objects, the illuminating direction for the virtual image can be corrected in accordance with the illuminating direction for the real image, and the image composition can then be performed. In this manner, a composed animation giving no unnatural feeling can be obtained. Also, in the cases of an image obtained by performing the image recording operation under the sunlight, by the estimation of the direction of the sunlight from the image, rear light compensation and image density compensation with respect to the image can be performed easily.
Particularly, the estimation of the direction of the light source, under which the image recording operation was performed on the object, should preferably be performed in accordance with the characteristics described below. Specifically, a calculation may be made to find a cosine between a direction vector directed from a virtual light source to a certain measuring point, which has been set at an area on the object image corresponding to the object area having uniform surface reflectivity, and a normal vector at the measuring point. In cases where the position of the virtual light source coincides with the position of the light source, under which the image recording operation was performed on the object, the relationship between the cosines, which have thus been calculated for a plurality of measuring points, and the intensity values of reflected light corresponding to the measuring points, i.e. the signal values of the object image signal, which signal values correspond to the measuring points, forms a straight line relationship. In accordance with such characteristics, the estimation of the direction of the light source, under which the image recording operation was performed, should preferably be performed in the manner described below. Specifically, after the position matching is performed on the object image signal and the three-dimensional shape signal, a plurality of measuring points are set at the area of the three-dimensional body image, which is represented by the position-matched three-dimensional shape signal that has been obtained from the position matching, the area of the three-dimensional body image corresponding to the area of the object, which area has approximately uniform surface reflectivity and which exhibits diffuse reflection. A calculation is made to find the cosine between the direction vector directed from each of the virtual light sources, which have been set in a plurality of arbitrary directions, to each of the measuring points. Also, a calculation is made to find the normal vector at each of the measuring points. The relationship between the calculated cosines and the signal values of the object image signal, which signal values correspond to the plurality of the measuring points, is then plotted with respect to each of the directions of the virtual light sources. Thereafter, the direction of a virtual light source, which direction is associated with a plotted relationship that forms an approximately straight line, is estimated as the direction of the light source, under which the image recording operation was performed.
Also, with the first light source direction estimating method and apparatus in accordance with the present invention, wherein the intensity of light radiated out from the light source, under which the image recording operation was performed, is also estimated, in cases where a real image and a virtual image are combined with each other in the manner described above, the virtual image can also be corrected in accordance with the intensity of light radiated out from light source, which intensity has been estimated from the real image, such that the virtual image is illuminated with light having an appropriate intensity, and the real image and the virtual image can then be combined with each other. Therefore, a composed image giving no unnatural feeling can be obtained.
Further, with the first light source direction estimating method and apparatus in accordance with the present invention, wherein the three-dimensional shape signal is the polygon signal, in cases where the plane, on which the measuring point is located, is found, the normal vector at the measuring point can be calculated immediately from the formula defining the plane. Therefore, the direction of the light source, under which the image recording operation was performed, can be estimated easily.
With the second light source direction estimating method and apparatus in accordance with the present invention, in cases where only the signal values of the object image signal and the relationship between the position of the three-dimensional body image, which is represented by the position-matched three-dimensional shape signal that has been obtained from the position matching with the object image signal, and the positions of virtual light sources having been set in the plurality of the arbitrary directions are found, the direction of each of the light sources, under which the image recording operation was performed, can be estimated. Therefore, it is not necessary for a measuring point, which allows calculation of an inverse matrix, to be found as in the technique proposed by Sato, et al. Also, it is not necessary for matrix operations to be performed. Accordingly, the directions of the plurality of the light sources, under which the image recording operation was performed, can be estimated with comparatively simple operations. Further, in cases where a real image and a virtual image, such as a CG image, are to be combined with each other, the virtual image can be corrected in accordance with the directions of the plurality of the light sources having been estimated from the real image, such that the virtual image is illuminated by appropriate light sources, and the real image and the virtual image can then be combined with each other. Therefore, a composed image giving no unnatural feeling can be obtained. Furthermore, in an encoding system, such as MPEG-4, wherein an image is transferred with respect to each of objects and image composition is performed at the stage of seeing an animation, in cases where a real image and a virtual image are contained in the objects, the illuminating direction for the virtual image can be corrected in accordance with the illuminating direction for the real image, and the image composition can then be performed. In this manner, a composed animation giving no unnatural feeling can be obtained.
Particularly, the estimation of the directions of the light sources, under which the image recording operation was performed on the object, should preferably be performed in accordance with the characteristics described below. Specifically, a calculation may be made to find a cosine between a direction vector directed from a virtual light source to a certain measuring point, which has been set at an area on the object image corresponding to the object area having uniform surface reflectivity, and a normal vector at the measuring point. In cases where the position of the virtual light source coincides with the position of one of the light sources, under which the image recording operation was performed on the object, the relationship between the cosines, which have thus been calculated for a plurality of measuring points, and the intensity values of reflected light corresponding to the measuring points, i.e. the signal values of the object image signal, which signal values correspond to the measuring points, forms a straight line relationship. In accordance with such characteristics, the estimation of the direction of each of the light sources, under which the image recording operation was performed, should preferably be performed in the manner described below.
Specifically, the position matching is performed on the object image signal and the three-dimensional shape signal, and combinations of virtual light sources are selected from the virtual light sources having been set in the plurality of the arbitrary directions. Also, the plurality of the measuring points are set at the area of the three-dimensional body image, which is represented by the position-matched three-dimensional shape signal that has been obtained from the position matching, the area of the three-dimensional body image corresponding to the area of the object, which area has approximately uniform surface reflectivity and which exhibits diffuse reflection. Further, the single-irradiation measuring points, which are exposed to only light radiated out from a single virtual light source when light is irradiated from each of the combinations of the virtual light sources to the three-dimensional body image, are selected from the plurality of the measuring points. Each of the single-irradiation measuring points is exposed to only the light radiated out from one virtual light source, which is among the virtual light sources constituting each combination, (in cases where a combination is constituted of only one virtual light source, the light radiated out from the one virtual light source). Furthermore, a calculation is made to find the cosine between the direction vector, which is directed from the single virtual light source to each of the single-irradiation measuring points, and a normal vector at each of the single-irradiation measuring point. Also, the relationship between the calculated cosines and signal values of the object image signal, which signal values correspond to the single-irradiation measuring points, is plotted with respect to the single virtual light source in each of the combinations of the virtual light sources. The direction of a single virtual light source, which direction is associated with a plotted relationship that forms an approximately straight line, is estimated as the direction of one light source among the light sources, under which the image recording operation was performed. Thereafter, the relative intensity value of light radiated out from the one light source is estimated in accordance with the inclination of the straight line, which is set by the plotted relationship utilized for the estimation of the direction of the one light source. In accordance with the relative intensity value of light radiated out from the one light source, the corrected object image signal is acquired, in which the effects of the light radiated out from the one light source have been removed. Also, with respect to the virtual light sources other than the virtual light source located in the direction of the one light source having been estimated, the selection of combinations of virtual light sources, the selection of single-irradiation measuring points in accordance with the corrected object image signal being employed as a new object image signal, the calculation of cosines, the plotting, the estimation of a direction of one light source, the estimation of a relative intensity value of light radiated out from the one light source, and the acquisition of a corrected object image signal, are iterated till directions of all of the light sources, under which the image recording operation was performed, are estimated.
Also, with the second light source direction estimating method and apparatus in accordance with the present invention, wherein the relative intensity values of light radiated out from the light sources, under which the image recording operation was performed, are also estimated, in cases where a real image and a virtual image are combined with each other in the manner described above, the virtual image can also be corrected in accordance with the relative intensity values of light radiated out from the light sources, which relative intensity values have been estimated from the real image, such that the virtual image is illuminated with light having an appropriate intensity, and the real image and the virtual image can then be combined with each other. Therefore, a composed image giving no unnatural feeling can be obtained. In cases where the surface reflectivity of the object is known, the absolute intensity of light radiated out from each of the light sources can also be estimated.
In such cases, with respect to an image having been obtained by performing the image recording operation under a plurality of different light sources, the effects of only a light source located in a specific direction can be compensated for. For example, in cases where the object image signal is composed of red (R), green (G), and blue (B) color signals, the intensity value of light radiated out from the light source may be estimated with respect to each of the color signals. In this manner, the ratio among the intensity values of the R, G, and B color components of the light radiated out from the light source can be calculated. Therefore, for example, it is possible to perform processing for compensating for the effects of only a light source radiating out light, in which the intensity of a specific color component (e.g., the R component) is higher than the intensities of the other color components.
In the second light source direction estimating method and apparatus in accordance with the present invention, in cases where the number of the light sources, under which the image recording operation was performed, is known, combinations of the virtual light sources in accordance with the number of the light sources, under which the image recording operation was performed, may be selected from the virtual light sources having been set in the plurality of the arbitrary directions. Therefore, in such cases, it is not necessary for the estimation of the directions of the light sources to be performed for the combinations with respect to the entire number of the light sources. Accordingly, the estimation of the directions of the light sources can be performed quickly.
Further, with the second light source direction estimating method and apparatus in accordance with the present invention, wherein the three-dimensional shape signal is the polygon signal, in cases where the plane, on which the measuring point is located, is found, the normal vector at the measuring point can be calculated immediately from the formula defining the plane. Therefore, the direction of the light source, under which the image recording operation was performed, can be estimated easily.